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EXECUTIVE SUMMARY 


The general purpose of this five-year evaluation is to assess the effectiveness of Milwaukee’s independent 
charter schools in promoting student achievement growth. Independent charter schools are authorized by non- 
school-district entities and are considered “independent” because they are not a part of the Milwaukee Public 
School District (MPS). Throughout the course of this report we will estimate four-year achievement gains for 
independent charter school students who were in grades 3-8 during the 2006-07 school year using reading and 
math achievement data from the Wisconsin Knowledge and Concepts Examination (WKCE). Specifically, the 
report presents the results of an analysis comparing achievement gains of independent charter students to the 
achievement gains of a carefully matched sample of students attending MPS. 

This report draws upon a panel of 2,295 students attending 10 of Milwaukee’s 14 independent charter schools 
who were in grades 3-8 in 2006-07 with test scores for that year. The four charter schools excluded from the 
sample either were not open for both the baseline and outcome years or did not enroll students in tested grades. 
The 2,295 tested Milwaukee independent charter school students were carefully matched to an identically sized 
sample of students attending MPS to provide a comparison group against which the achievement gains of 
independent charter students could be assessed. Students were matched on prior achievement and propensity 
scores, which help to control for differences between students on observable characteristics. We are confident our 
matching algorithm produced a charter and MPS sample equivalent on prior achievement. However, similar to 
other observational studies, our study is unable to control for all potential sources of unobservable selection bias. 
We believe this potential threat is less of a concern because highly motivated parents in MPS, similar to parents 
of charter school students, have many alternative options to exercise choice within the Milwaukee Public School 
system. 

The results of the comparisons on math and reading scores differ by years and by the statistics employed. We 
reported last year that there was: 

...a clear pattern of positive charter school effects growing over time. There was little consistent 
evidence of differences in achievement gains between charter and MPS students after one year. 

The second year growth was better for charters in some models and for some tests, but not for 
others. In the third year of growth, a sizable independent charter school advantage was apparent in 
all of our analyses (Witte et al., 2011). 

That trend was not continued in the fifth year where estimates of four-year achievement growth are positive 
for charter schools but the basic models do not produce statistically significant differences between students 
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attending all independent charters and the MPS sample of students. On the other hand, students in conversion 
charters schools, which were once private schools, consistently outperformed similar MPS students in the 
matched sample in every year. In one model, the larger growth in math after five years is positive but not quite 
statistically significant. 

One of our supplementary analyses determined that students who remained in charter schools over five years 
(e.g. “stayers”) made significant achievement gains in both reading and math compared to their counterpart 
stayers in MPS. This finding holds for charter schools on average, as well as for conversion and non-conversion 
charter schools. These results are between three- and four- tenths of a standard deviation and significant at the 
99 percent confidence level. 

This project has been funded by a diverse set of philanthropies including the Annie E. Casey, Joyce, Kern Family, 
Lynde and Harry Bradley, Robertson, and Walton Family foundations. We thank them for their generous 
support and acknowledge that the actual content of this report is solely the responsibility of the authors 
and does not necessarily reflect any official positions of the various funding organizations, the University of 
Arkansas, or the University of Wisconsin. We also express our gratitude to officials at MPS, the independent 
charter schools, and the state Department of Public Instruction for their willing cooperation, advice, and 
assistance. We are appreciative of the constructive comments on a preliminary draft from outside experts as well 
as the School Choice Demonstration Project Research Advisory Board and research team. All remaining errors 
are the responsibility of the authors alone. 
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Charter Schools 

Charter schools are tuition-free public schools that are authorized to operate within an agreed “charter.” 
Charters often specify the size of the school, its mission, specialized curricula and pedagogy, unique personnel 
practices, and specific goals that the school must meet over time in order to be reauthorized. Most charter 
schools use an open enrollment system that permits students to attend the school even if they do not live close 
by. Thus, charter schools are subject to parental school choice. To facilitate these unique schools, they are often 
given waivers from some of the administrative and accountability requirements of other public schools. This 
does not exempt charter schools from the testing and reporting requirements of the federal No Child Left 
Behind law. 

Since the opening of the first charters schools in the US in the early 1990s, the number of charter schools has 
dramatically increased. In 2010, over 5,000 charter schools serving close to 1.7 million students operated in 40 
states and the District of Columbia (Center for Education Reform, 2011). Following the recent “Race to the 
Top” initiative, which requires states to relax laws restricting the creation of charter schools, we can expect to see 
even further expansion of charter schools in the coming years. 

Similar to national trends, the number of charter schools in Wisconsin has grown widely, from 17 in 1997 
to 206 in 2010 (Evers et al., 2010). Charter schools in 2010 served 37,000 students in the state (School 
Management Services, 2011). Government officials see the potential of charter schools as part of a reform to 
transform public education in the state. For the first time, Governor Jim Doyle and State Superintendent Tony 
Evers attended the Wisconsin Charter Schools Conference in April, 2009 (Borsuk, 2009). In October of 2009, 
President Barack Obama and Education Secretary Arne Duncan visited Wright Middle School, a charter 
school in Madison, to highlight the role of charter schools in the “Race to the Top” initiative. Wisconsin has also 
received $86 million in federal funding over the next five years to support charter schools in Milwaukee and the 
state by allocating grants to new and existing charter schools. 

In Milwaukee, charter schools are one among a wide variety of school choice options including charter and 
magnet schools affiliated with MPS, open enrollment into other public school districts, and private schools 
accepting vouchers under the Milwaukee Parental Choice Program. In 2006-07, charter schools in Milwaukee 
comprised close to a quarter of the charter schools in the state. 

This longitudinal study evaluates the impact of independent charter schools on student achievement in 
Milwaukee, Wisconsin over five years. Milwaukee is one of the few places in the U.S. that contains both district- 
authorized charter schools and independent charter schools. As of 2006-07, 38 district-authorized charter 
schools were part of the Milwaukee Public School system. Of these 38 district-authorized charters, a total of 
25 were staffed by teachers who remain employees of the school district and bound by the union-negotiated 


Milwaukee Independent Charter Schools Study: Final Report on Four- Year Achievement Gains 


2 


February 2012 


collective bargaining agreement 

(Table 1). These schools are referred Table 1. Types of Public Charter Schools in Milwaukee, Wl, 2006-2007 

to as “instrumentality” charters. 

The remaining 13 MPS “non- 
instrumentality” charter schools are 
permitted to hire and employ non- 
union teachers. This study does not 
evaluate district-authorized charters. 

Independent charters are a distinctive 
type of charter school in Milwaukee. 

They were created by 1997 legislation 
to be authorized by the City of 
Milwaukee Common Council, the 
University of Wisconsin-Milwaukee 
(UWM), the Milwaukee Area 
Technical College, or the University of Wisconsin at Parkside (Racine). They are not connected to MPS, and 
they are non-instrumentality charter schools (hence they are not unionized). Of the 9 UWM and 5 City of 
Milwaukee independent charters open when this study began in 2006-07, 10 are the subjects of this research. 
Table 2 presents the student enrollments by grade for the baseline year of 2006-07 for our school sample and 
an explanation for excluded schools. As is apparent, UWM charter schools have many more students than City 
charters, and there are very few students in grade 9 compared to grades 3 to 8. 1 


Type 

Number 

Percentage of All 

MPS Instrumentality 

25 

48.1 

MPS Non-Instrumentality 

13 

25.0 

MPS Total 

38 

73.1 

Independent U ofW-Milwaukee 

9 

17.3 

Independent City of Milwaukee 

5 

9.6 

Independent Total 

14 

26.9 


Source: Wisconsin Charter Schools Yearbook 2006-2007 
http://dpi.state.wi.us/sms/pdf/2006-07yearbook.pdf 


Table 2. Milwaukee Independent Charter School Sample Enrollment, 2006-07 


Grade 

Schools 

3 

4 

5 

6 

7 

8 

9 

TOTAL 

UWM 

6 

328 

331 

338 

287 

241 

239 

140 

1904 

City 

4 

92 

89 

99 

88 

119 

80 

58 

625 

TOTAL 

10 

420 

420 

437 

375 

360 

319 

198 

2529 


Source: Charter Schools page on the Department of Public Instruction website: http://www.dpi.state.wi.us/sms/xls/0607enrl.htm 

Note: For the 2006-07 baseline year, there are no test score data for structural reasons for the following schools: Inlands Sea School 
of Expeditionary Learning (ISSEL), Milwaukee Renaissance Academy (MRA), Seeds of Flealth (SoFiE), School for Early Development 
(SEDA), and Massai Institute which has closed as of 2007-08. For the first four schools, they did not test in November 2006 when 
schools typically test because they were not yet open. ISSEL opened in 01/2006, MRA in 08/2007, and SoHE in 08/2007. SEDA is 
an early education school with grade levels K4-2 and does not have data for grades 3-8, or grade 10 because it does not have these 
grade levels at its school. Bruce Guadalupe Community School transitioned from the oversight of MPS to being authorized as a charter 
by UWM in 2009-2010. In addition, in the 2010-2011 school year, the City chartered King’s Academy while UWM chartered Urban 
Day School, Veritas High School and a new campus of Milwaukee College Preparatory School, Lindsay Fleights. Many of the students 
attending Lindsay Fleights formerly attended the Academy of Learning and Leadership which closed in September 2010. 


1 Ninth grade students are not included in this study because tests were not given in the ninth grade. 
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Research on Charter Schools 

Supporters see the potential of high-quality charter schools to help transform the education system by raising 
achievement levels, closing achievement gaps, placing competitive pressure on traditional public schools and 
stimulating greater innovation. They posit that giving charter schools more flexibility over such practices as 
hiring teachers, budgeting school funds, and selecting curricula will lead to these positive outcomes (Finn, 
Manno, & Vanourek, 2001; Payne & Knowles, 2009). Further, through a system of accountability, they expect to 
reduce the number of low-quality charter schools that are not able to meet the standards they agreed to in their 
charters. 

In contrast, critics are concerned about charter schools drawing away resources from traditional public schools 
(e.g. teachers, funding, and motivated students), increasing racial segregation, and lacking the accountability 
structure to close or improve low-quality charter schools (Wells et al., 2002). They fear charters are performing 
no better and sometimes worse than traditional public schools. To date the research on the performance of 
charter schools is somewhat mixed, ranging from negative, neutral, mildly positive, to a few specific studies 
which are strongly positive (Bifulco & Ladd, 2006; Sass, 2006; Ballou et al., 2006; Hanushek et al., 2007; Booker 
et al., 2007; Zimmer et al., 2009; Witte et al., 2007; Witte & Lavertu, 2009; CREDO, 2009; Hoxby et al., 2009; 
CREDO, 2010; Abdulkadiroglu et al., 2009; Tuttle et al., 2010; Gleason et al., 2010). 

Most of these prior studies are observational, with a smaller number of studies employing randomized 
experimental designs. While randomized experiments are recognized as producing gold standard results, they 
are costly and logistically less feasible than securing existing longitudinal data used in observational studies. In 
general, where there have been positive or negative impacts of charter schools, they have typically been small 
in magnitude (Hill et al., 2006). The exceptions to this general trend are some more recent randomized trials of 
charter schools in Boston and New York (Abdulkadiroglu et al., 2009; Hoxby et al., 2009) and an observational 
study of those charters affiliated with the Knowledge is Power Program (KIPP) (Tuttle et al., 2010) which find 
strong positive charter impacts. Research on charter impacts in Milwaukee more closely resembles the general 
trend of charter school performance, with two prior observational studies showing modestly positive gains in 
math for charter schools authorized by the Milwaukee Public Schools (Witte et al., 2007; Witte & Lavertu, 
2009). Our report of one year student achievement growth of students in Milwaukee independent charter 
schools found little difference in achievement between these schools and MPS schools (Witte et al., 2010). 
However, the report highlighted that students in conversion charter schools - i.e. formerly private schools - did 
have more positive achievement growth than students in traditional MPS schools. Our analysis of two- and 
three-year growth of Milwaukee charter students compared to MPS students, also released previously, reported 
a charter school advantage that was somewhat larger and more consistent than the results of the initial analysis 
(Witte et al., 2011). 

In this report, we extend our previous charter school studies and evaluate four-year achievement gains. Using 
five years of panel data — 2006-07 to 2010-11 — we estimate models of achievement gains for charter school 
students who were in grades 3 through 8 at baseline (2006-07), relative to similar students in MPS, controlling 
for baseline test scores and student characteristics. 
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Research Question 

Through this evaluation we endeavor to understand whether students benefit in the short term and the long 
term from attending an independent charter school. In this report the primary research question is: Do 
Milwaukee’s independent charter schools produce greater four-year achievement gains than Milwaukee public 
schools? For purposes of this study, achievement is measured by performance on the reading and mathematics 
sections of the Wisconsin Knowledge and Concepts Examination (WKCE) that all public school students are 
required to take in grades 3 to 8 and 10 (Wisconsin Department of Public Instruction, 2011b). The WKCE is 
administered in the fall of each school year and uses short answer and multiple choice questions to test student 
mastery in reading, math, language arts, science and social studies. Scores on these examinations are recorded in 
both scale (or developmental) scores and proficiency levels. We rely on scale scores in this analysis. As indicated 
below, we standardize these scale scores in order to allow comparisons across grade levels. Student test scores 
and demographic characteristics used in this study were provided by the Office of Research and Evaluation at 
MPS, the Office of Charter Schools at the University ofWisconsin-Milwaukee, and the City of Milwaukee 
Common Council. 2 

Matched Samples 

The first step in our analysis involved constructing the comparative samples of students. Because the total 
number of students in independent charters for which test scores were available in 2006-07 was 2,295, we 
decided to include all of those students in the charter school sample. The issue was then how to create a relevant 
matched sample that would be similar on important observed characteristics at baseline. To do that we first 
selected a random sample of MPS students matched by grade. In doing so we discovered that the baseline test 
scores (November 2006) for that group differed from those in the independent charter schools in a number 
of grades. The random MPS sample of students usually scored higher than the independent charter students. 
Thus, if we had used the random sample we would have started out with students at different levels of prior 
achievement. 

To adjust for this problem we undertook a two-step procedure. 3 First, each student in the charter sample was 
grouped with the complete set of MPS students in their grade with baseline WKCE test scores within five 
percent of their score. This was done within 20 bands from the lowest to the highest based on the distribution 
of independent charter student combined reading and math test scores. As an example, the first band consisted 


2 We are particularly grateful to Deb Lindsey of MPS, Robert Kattman of UW-M, and Cindy Zautcke of the Common Council for 
their support and assistance in obtaining the necessary data. All student data were provided to us absent personal information 
about the student, such as name and address, or such "personal signifiers"were deleted from the data prior to analysis. 

3 See Witte etal., 2010a for figures and tables describing the results of our matching protocol. 
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of students who scored in the lst-5th percentiles, the second band consisted of students in the 6th-10th 
percentile, etc. Second, the charter panelist and each MPS student within that five percent test score band 
were assigned a propensity score that predicted their likelihood of being in a charter school based on race, 
gender, English Language Learner (ELL) status, and participation in the federal Free/Reduced Price Lunch 
(FRL) program. The MPS student within the grade band with the charter school propensity score closest to 
the propensity score of a given student in the charter panel was drawn out of the panel (without replacement) 
and became a member of the MPS comparison sample. The result of all these matches was a sample of 2,295 
independent charter school students and 2,295 MPS comparison students that closely resemble the charter 
school students on baseline test scores and other factors that predict charter school enrollment. The purpose of 
this procedure was to reduce the differences in observed characteristics that existed between the independent 
charter students and a random sample of MPS students. 

When comparing the Independent Charter and MPS Matched samples, the only statistically significant 
differences in baseline test scores occur in 4th grade math and 6th grade math. Both of those differences 
between the charter and matched samples are statistically significant only at the 90 percent confidence level, the 
lowest confidence level that we use in this evaluation. This suggests the matching was successful. Thus, in terms 
of prior achievement we have created matched samples that are essentially equivalent at baseline. 

Our matching algorithm also produced charter and MPS student samples that are generally similar regarding 
other important measurable student characteristics (see Appendix C and Table C2 for details). The two samples 
are very close on race and gender demographics. The MPS Matched sample does differ significantly from the 
Independent Charter Sample regarding populations of exceptional education 4 and free lunch students, though 
a random sample of MPS students would have differed from the charter sample even more regarding these two 
student characteristics (See Witte et al., 2010). Because of these differences, in most of the analyses to follow, we 
independently control for all of these student characteristics in our most precise regression models. 5 

The matching design and baseline control variables limit the extent to which measurable student characteristics 
might bias our analysis of independent charter and MPS student test score gains. Because students were not 
randomly assigned to the two groups, however, we cannot rule out unmeasured student characteristics as a 
potential source of bias. For example, if the students in Milwaukee independent charter schools are similar to 
our matched MPS sample in most ways except that they have more motivated parents, as demonstrated by the 


4 We ran a sensitivity analysis predicting the likelihood of a student having an exceptional education status. The fixed effects 
regression results showed that the charter coefficient was not significant meaning that students in both sectors had an equal 
likelihood of having a disability. 

5 The initial difference between the charter and matched sample on free lunch status is due to incomplete free lunch data counts 
in a few schools. We correct for this in our models in two ways. If a student had a free lunch observation in 2007-08, 2008-09, 
2009-10 or 2010-1 1, we back-filled the data. In addition, for students with missing data on free-lunch or any other control 
variable, we include an indicator in our models controlling for this missing data. Doing so allows our regression models to draw 
upon the actual data in each student observation, and only that actual data, to inform the coefficient estimates of the model 
(Cohen & Cohen, 1 983). 
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fact that they enrolled the student in a school of choice, then the charter students might demonstrate stronger 
achievement gains simply due to such a “self-selection” bias. On the other hand, if parents seek alternatives to 
their neighborhood public school primarily when their child is struggling or exhibiting behavior problems, the 
match on baseline achievement might not fully capture the inherent educational disadvantages of charter school 
students, thereby biasing our analysis against better performance from charter schools. 

We think that the fact that our study is situated in Milwaukee helps to reduce the threat of positive or negative 
unmeasured selection biases. As discussed above, many school choice options are available to parents even 
within the Milwaukee Public School (MPS) system. Highly motivated parents, or parents of students who are 
struggling in their neighborhood public school, can and likely do seek out alternative placements for their child 
within MPS. These options include vouchers to attend private schools, magnet schools, MPS charter schools, 
and open enrollment into other school districts. Since school choosers are present in both our charter and MPS 
matched comparison samples, concerns about self-selection bias when comparing student achievement gains 
across sectors are, to some extent, mitigated. 

Main Effects and Supplemental Analyses 

The primary goal of this report is to estimate the effect of independent charter school attendance on student 
achievement gains. However, a secondary goal of the report is to understand the specific mechanisms through 
which any effects might operate. Over the five years of this study, students in the MPS and independent charter 
samples likely experienced a number of changes in family and school context, and some of these changes are 
likely to affect student achievement. An example we explore in the appendix is school switching. In this study, 
and many other studies, school switching has a negative effect on student achievement (e.g. Hanushek, Kain & 
Rivkin, 2004; Cowen et al., 2010). If there are differences in the rate of switching schools for charter school 
and MPS students, that may contribute to an explanation of the main effect which is the basic comparison 
on achievement growth between the two samples. Thus in what follows we present main effects based on all 
students in the samples, and then provide a set of supplemental analyses which refine and help us understand 
important variations in and possible explanations for our main effects. 

For the main effects in this report students are assigned to their sector (charter or MPS) based on where they 
were enrolled in 2006. If they cross over to another sector and we can locate a test score for them, we use 
that score and keep them in the study still assigning them to their initial sector. This is standard practice in 
randomized clinical trials. In some of the supplemental analyses we relax this condition. 
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MAIN EFFECTS ON STUDENT ACHIEVEMENT GAINS: 2006-07 to 2010-11 


We employ both descriptive statistics and multivariate regression methods to compare four-year achievement 
gains of students in independent charter schools and comparable, matched students in Milwaukee Public 
Schools. Prior to any analysis, we first standardized the WKCE scale scores into z-scores using the MPS district 
means and standard deviations for math and reading. 6 For all MPS students this procedure would produce an 
average z-score of 0 with a standard deviation of 1.0. Our samples deviate from these norms at baseline in that 
the average standardized student score is below 0, a fact which further confirms that the students in both the 
independent charter sample and MPS matched sample are educationally disadvantaged relative to the average 
MPS student. These normalized z-scores are used throughout the analysis. 

Average Math and Reading Achievement Growth 

We compared mean growth in standardized scores for independent charter and MPS students across grades and 
subjects. One-, two-, three-, and four-year change in math and reading achievement for students in grades 7, 8 
and 10 for the sample of students remaining in 2010 are presented in Tables 3 and 4 for our charter and MPS 
samples. 7 These results are broken out by grade level to examine the variation in student learning gains by sector 
across the different grades. The change calculations are created by using the student’s grade in 2010, where, for 
example, students in grade 3 at baseline are in grade 7 in 2010. Using a student’s grade in 2010, we estimate the 
mean change score for charter and MPS students in that grade. Since 2010 is the outcome year, a one-year change 
is defined as subtracting the 2009 score from the 2010 score. Similarly, a two-year change is computed by taking 
the difference of the 2010 score and the 2008 score, etc. After finding the average change score for each sector, 
we take the difference between the two averages to determine whether growth favors charter or MPS students. 

The grade-specific results, which are presented in Table 3 for math and Table 4 for reading, are generally a mix 
of non-significant and positive results favoring charter schools. The clearest charter advantages are for students 
in grade 7, with these charter students making significantly greater growth in math and reading in all but the 
first year. Likewise, charter school students in grade 8 exhibit greater three- and four-year growth in reading 
(Table 4). Across all grades, there are charter advantages in both reading and math. Charter school students 
make greater two- and four-year growth in math, as well as greater two-, three- and four- year growth in 
reading. The remaining results are non-significant, meaning that the performance of charter school students 
does not statistically differ from that of similar students in MPS. 


6 We computed normalized z-scores by grade level in all four years for reading and math. For example, the formula for 
ZMath2007 in Grade 3 was ((Grade 3 ScaleMath2007- Grade 3 MPS district mean scale score)/(Grade 3 MPS district standard 
deviation)). 

7 The number of grades and the sample sizes are considerably reduced from our baseline sample because students "aged out" 
of the sample, advancing to grades for which there were no test scores. The sample in this report thus consists of students 
originally in grades 3, 4 and 6 in 2006 for whom we could locate a test score in 2010. 
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There are exceptions to the general trend outlined above. Specifically, charter school students in grade 10 make 
less two-year growth in reading, and three-year growth in both reading and math. Eighth grade charter school 
students exhibit lower one-year growth in math compared to MPS students. This estimate, however, only reaches 
the lowest level of statistical significance. Comparing all grades together, there are no statistically significant 
differences in math, but there are three- and four-year advantages for charter school students in reading. 


Table 3. Mean Math Achievement by Grade in Outcome Year, 2006-07 to 2010-11 


Grade 

2010 

Group 

One-Year 

Change 

(09-10) 


Two-Year 

Change 

(08-10) 


Three-Year 

Change 

(07-10) 


Four-Year 

Change 

(06-10) 




Mean 

Growth 

s.e.(diff) 

Mean 

Growth 

s.e.(diff) 

Mean 

Growth 

s.e.(diff) 

Mean 

Growth 

s.e.(diff) 

7 

Charter 

.085 


.264 


.258 


.372 



MPS Matched 

.026 


.071 


.006 


.165 



(Difference) 

(.059) 

(.059) 

(.193)*" 

(.069) 

(.252)"' 

(.075) 

(.207)'" 

(.077) 


Sample Size 

[N=492] 


[N=486] 


[N=487] 


[N=518] 


8 

Charter 

-.048 


-.010 


.092 


.363 



MPS Matched 

.058 


.069 


.145 


.286 



(Difference) 

(-.106)' 

(.055) 

(-.079) 

(.059) 

(-.053) 

(.067) 

(.077) 

(.073) 


Sample Size 

[N=528] 


[N=529] 


[N=531] 


[N=569] 


10 

Charter 



-.088 


-.152 


-.178 



MPS Matched 



-.081 


.032 


-.042 



(Difference) 



(-.007) 

(.079) 

(-.184)" 

(.075) 

(-.136) 

(.088) 


Sample Size 



[N=294] 


[N=373] 


[N=422] 


All Grades 

Charter 

.019 


.074 


.081 


.201 



MPS Matched 

.043 


.019 


.062 


.128 



(Difference) 

(-.024) 

(.040) 

(.054) 

(.039) 

(.020) 

(.042) 

(.072) 

(.047) 


Sample Size 

[N=1020] 


[N=1309] 


[N=1391] 


[N=1509] 



***p<0.01, **p<0.05, *p<0.10 

Note: The table presents one-, two-, three- and four-year change for students in grades 7, 8, and 10 in 2010. Two sample t-tests 
were run to test the significance of differences in average growth between our MPS Matched sample and Charter sample. Mean growth 
scores for each sector are rounded to the third significant digit. Response weights were included in the estimation of differences in 
means. 
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Table 4. Mean Reading Achievement by Grade in Outcome Year, 2006-07 to 2010-11 


Grade 

2010 

Group 

One-Year 

Change 

(09-10) 


Two-Year 

Change 

(08-10) 


Three-Year 

Change 

(07-10) 


Four-Year 

Change 

(06-10) 




Mean 

Growth 

s.e.(diff) 

Mean 

Growth 

s.e.(diff) 

Mean 

Growth 

s.e.(diff) 

Mean 

Growth 

s.e.(diff) 

7 

Charter 

.032 


.259 


.246 


.311 



MPS Matched 

.005 


.072 


.030 


.131 



(Difference) 

(.027) 

(.058) 

(.187)*** 

(.071) 

(.216)*** 

(.080) 

(.180)** 

(.090) 


Sample Size 

[N=492] 


[N=486] 


[N=487] 


[N=518] 


8 

Charter 

.006 


.088 


.263 


.302 



MPS Matched 

.009 


.044 


.103 


.160 



(Difference) 

(-.003) 

(.054) 

(.044) 

(.058) 

(.160)*** 

(.062) 

(.142)** 

(.069) 


Sample Size 

[N=529] 


[N=531] 


[N=530] 


[N=571] 


10 

Charter 



-.145 


-.145 


-.064 



MPS Matched 



.010 


.007 


-.007 



(Difference) 



(-.155)** 

(.078) 

(-.152)“ 

(.074) 

(-.057) 

(.080) 


Sample Size 



[N=294] 


[N=373] 


[N=423] 


All 

Charter 

.019 


.094 


.137 


.191 


Grades 

MPS Matched 

.007 


.042 


.046 


.089 



(Difference) 

(.012) 

(.040) 

(.053) 

(.040) 

(.091)** 

(.042) 

(.101)** 

(.046) 


Sample Size 

[N=1021] 


[N=1311] 


[N=1390] 


[N=1 512] 



***p<0.01, **p<0.05, *p<0.10 

Note: The table presents one-, two-, three- and four-year change for students in grades 7, 8, and 10 in 2010. Two sample t-tests were run 
to test the significance of differences in average growth between our MPS Matched sample and Charter sample. Mean growth scores for 
each sector are rounded to the third significant digit. Response weights were included in the estimation of differences in means. 


The Distribution of Math and Reading Growth 

It is equally important to understand the distribution of growth as it is to note averages. The density, which 
approximates the number of individuals at specific points in the achievement distribution, is provided in kernel 
diagrams in Figures 1 and 2. Examining the kernel density distributions of both sectors in reading and math 
will help us to understand the shape of the growth in our samples. For both reading and math, the distributions 
are mostly overlapping and are normally distributed. Non-overlapping distributions would indicate variation 
in performance between the two sectors. For math, the MPS matched sample has somewhat more students 
just below the mean of the distribution, while the reverse is true for charter students just above the mean. In 
reading (Figure 2) the peak of the charter distribution is to the right of the MPS distribution demonstrating 
an advantage for charter students. The range of students scoring at the high end and low end is very similar in 
both sectors. 
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Figure 1. 

Math Growth (Z-Scores) 

for All Students in Grades 7, 8, and 1 0: 2006 to 2010 


Math Growth 2006-2010 



Figure 2. 

Reading Growth (Z-Scores) 
for All Students in Grades 7, 8, and 1 0: 2006 to 201 0 


Reading Growth 2006-2010 



Models for Math and Reading Achievement Gains 

Using the dataset described above, we estimate the impact of independent charter school attendance on gains 
in math and reading controlling for student characteristics. Our analytic sample used to estimate the effect of 
independent charter school attendance on four-year achievement gains consists of 1,559 students. To control for 
potential achievement differences by grade, we include grade indicator variables in all equations. We control for 
baseline achievement by including the student’s baseline (2006) test scores in both subjects. The basic model for 
estimating four-year gains is represented by the following equation: 


(eq 1) Y 2010 = P 0 + P 1 C i +P 2 Y aBOfaii +P 3 Y 2006r , ,+PA + P 5 X + a 

In this equation, for each student i, pj represents the effect of student enrollment in a charter school in 2006-07 
(C=l) and p, and P 3 estimate the impact of baseline math and reading achievement. With this specification, the 
contribution of the baseline test to the estimate of the 2010 test score is unconstrained in that p, and p, can take 
any value. 8 p 4 represents a vector of grade-specific contributions to the intercept and p- represents the impact of a 
set of student-level characteristics, X , such as gender and race/ethnicity. 

The outcomes of interest are 2010-11 reading and math scores on the WKCE which are standardized for 
students in grades 7, 8 and 10 in 2010. Student characteristics included are those typically found in studies of 
charter school performance and include free and reduced lunch status, exceptional education status (ExEd), 


8 Some researchers have used differences in test scores as the dependent variable by subtracting the baseline test score from 
the outcome year test score. Flowever, if we want to model achievement growth controlling for prior achievement, this has 
the effect of constraining the effect of prior achievement to 1.0, which empirically is not the true parameter. Thus, we favor the 
estimation model in Equation 1. 
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race, and gender. 9 English language learner (ELL) status was not included because there were very few students 
formally classified as ELL in the charter schools. The race indicator variable is coded as 1 for black students 
and 0 for non-black students, which serves as the reference group. We collapse racial groups other than black 
students into the non-black students category because there are substantially fewer whites, Hispanics, Asians, 
and Native Americans in the sample. 10 To correct for potential asymmetric attrition from the charter and MPS 
samples, we use non-response weights that were constructed using observable baseline student characteristics. 
The weighted results reported below are substantively similar to the results from estimating the same models on 
unweighted samples. The estimations also account for the clustering of students within schools by employing 
robust and clustered standard errors. 


Results for Regression Models of Charter Impacts on Four-Year Math and Reading Achievement Growth 

Main effects of independent charter school status on four years of growth are presented in Table 5 in Models 
1 through 3. Model 1 controls only for charter status and shows the effect of charter attendance at baseline on 
student achievement in 2010. In 2010, students in independent charter schools score higher on average than 
MPS students in both reading and math, but the results are not statistically significant. This model does not tell 
us how much charter school students have gained because it does not control for prior achievement at baseline. * 11 
Model 2 controls for baseline (2006) achievement in both reading and math. The coefficients slightly increase 
for students attending charter schools, but the results are also not statistically significant. 

Of the models presented, Model 3 is our best model of student achievement gains because in this model 
we control for prior achievement and any differences in student characteristics. Our discussions of main 
effects and the supplementary analyses below will focus primarily on that specific model. As with Models 1 
and 2, the effect of charter school enrollment on reading and math scores from Model 3 is positive but not 
statistically significant. 


9 We acknowledge that participation in the federal free and reduced lunch (FRL) program can be an inconsistent measure of 
student poverty across grades and sectors, since older students are more likely to decline participation even if eligible and 
some schools outside of the traditional public school system choose not to participate in the program at all. Such problems 
are especially acute when comparisons are being made between the public and private school sectors (Peterson & Llaudet, 
2006). In spite of these concerns, we use the FRL indicator in our models for three reasons. First, our comparison is between 
different types of public schools — independent public charters compared to schools within a traditional public school system. 
Second, all of the schools represented in our sample do participate in the FRL program. Third, participation in the FRL program 
is the only proxy measure of student poverty available to us. Leaving the federal lunch program out of our models would have 
invited omitted variable bias of an unknown direction and magnitude to undermine our analysis. 

10 In 201 0-1 1 , there are 2,880 non-missing black students and 98 non-missing non-black students. 

1 1 If this were a randomized field trial, or if the initial matching on prior tests was perfect. Model 1 would also accurately estimate 
growth. However, because of the differences between samples in prior achievement in some grades (Table Cl) and in student 
characteristics (Table C2), more complex models will improve the accuracy of estimated effects. 
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Table 5. Four- Year Growth Models of Math and Reading Achievement for Average 

Charter Impacts, 2006-2010 



Model 1- Charter Status 

Model 2- Controlling for Prior 
Test Scores Only 

Model 3- Student 
Characteristics Included 

Math 

2010 

Reading 

2010 

Math 

2010 

Reading 

2010 

Math 

2010 

Reading 

2010 

Charter 2006 

.107 

(. 142 ) 

.047 

(. 148 ) 

.111 

(. 067 ) 

.069 

(. 083 ) 

.058 

(. 052 ) 

.053 

(. 066 ) 

2006 Score - 
Reading 



. 223 *** 

(. 029 ) 

. 466 *** 

(. 034 ) 

. 193 “* 

(. 026 ) 

. 41 1 *“ 
(. 026 ) 

2006 Score- 
Math 



. 432 *** 

(. 027 ) 

(. 025 ) 

. 445 “* 

(. 030 ) 

. 224 *“ 

(. 026 ) 

Grade 7 





-. 239 “ 

(. 108 ) 

-. 275 *** 

(. 088 ) 

Grade 8 





-.215 

(. 134 ) 

-. 234 “ 

(. 089 ) 

Grade 10 





-. 479 “* 

(. 137 ) 

-. 456 "* 

(. 105 ) 

Black 





-. 288 “* 

(. 079 ) 

-.399 " 
(. 080 ) 

Female 





-.067 
( . 068 ) 

.083 

(. 057 ) 

ExEd 





-. 218 “ 

(. 085 ) 

- 370 ". 

(. 075 ) 

Free Lunch 





-.076 

(. 056 ) 

-. 097 * 

(. 057 ) 

Constant 

1 3 *| *** 

(. 045 ) 

-.049 

(. 047 ) 

.017 

(. 030 ) 

. 070 “ 

(. 028 ) 

. 748 “* 

(. 134 ) 

833 *,, 

(. 109 ) 

N 

1556 

1559 

1556 

1559 

1556 

1559 

R 2 

.003 

.001 

.427 

.449 

.462 

.487 

F 

0.58 

0.10 

362.26 

318.54 




***p<0.01, **p<0.05, *p<0.10 


Tabled results control for race, gender, grade, free lunch, exceptional education in Model 3. Reference categories for these variables are 
non-black, male, grade 6 in 2010, no free lunch, and no exceptional education. Two prior test scores are also controlled for in Models 
2 and 3. Robust standard errors are estimated to account for the clustering of students within particular schools. F-tests cannot be 
computed using robust standard errors when a cell in the regression matrix is only a single student. 
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In all models estimating achievement gains, charter school students perform similarly to MPS students 
statistically in both reading and math (Table 5). Control variables in Model 3 of Table 5 perform as expected, 
giving us confidence in the reliability of our analysis. Consistent with prior research, students in both MPS and 
the independent charter sectors with higher baseline achievement (2006 test scores) have higher achievement 
four years later, relative to students with lower prior achievement. In contrast, black students in both sectors 
show lower four-year achievement gains than non-black students in most models. Similarly, students who 
receive exceptional education services exhibit lower achievement gains compared to non-disabled students. 

Figure 3 puts these results in historical context by providing a graphical depiction of the effect of independent 
charter attendance on yearly achievement gains in both math and reading. The early year results are based on 
Model 3 estimates from prior years as reported earlier (Witte, et al., 2010; Witte, et al, 2011). The dots represent 
the coefficient values in Model 3, and the bars represent the confidence intervals around those estimates. If the 
bars do not cross the zero line, we can reject the null hypothesis that there is no difference between independent 
charter and MPS achievement growth for the relevant year. As is apparent, all the mean estimates favor 
independent charters, but only the second and third years of growth in reading and the third year of growth in 
math allow us to say with at least 90% certainty that the independent charter advantages are greater than zero. 
Readers should note that the confidence intervals around the 2010 growth effect estimates are much wider than 
the intervals around the 2009 estimates. The main reason for this difference is the fact that a large cohort of 
students in the study, those who were in 7th grade in 2006, contributed test scores to the 2009 growth analysis 
but had “aged out” of the study sample for purposes of the 2010 analysis. All else equal, confidence intervals 
grow as sample sizes shrink. 


Figure 3. Effect of Independent Charter Attendance on Student Achievement, by Year 

Point Estimate and 90% Confidence Interval 
Math Reading 
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SUPPLEMENTAL ANALYSES OF STUDENT ACHIEVEMENT GAINS 


As stated in the introduction, this report is divided into main effects and supplemental analyses that help explain 
and understand the primary results. In this section we address three factors that add to our understanding of 
the variance in the primary effects. They are the type of independent charter school, charter school effects across 
the achievement distribution, and the effects for a subgroup of each sample - those who stayed in either charter 
schools or MPS schools for the full five years. A final analysis of students who switch schools is included in 
Appendix B. 

Variation in Main Effects by Type of Charter Schools 

In addition to understanding the main effect of attending an independent charter or a MPS school, we are 
also interested in the effects of two different types of charter schools. Four of the independent charter schools 
were initially private schools that changed school sectors by converting to independent charters (i.e. conversion 
charters). The other 6 charters were startup schools (i.e. non-conversion charters). 12 There were only 204 
students from our original baseline sample in 2010-11 enrolled in conversion charter schools, compared to 463 
students in the non-conversion charter schools. We capture and test for the differential effects of these two types 
of charter schools in the estimation of equation 2. 


(eq 2) 


Y^rPo + P.CC+P.NCC,- 


P 3 y 


+B Y 

2006m, i ^4 2« 


, +p G + p X + s. 

6r, l ' 5 l r 6 l l 


In this specification we split the charter indicator variable in equation 1 into conversion charters (CC) and non- 
conversion charters (NCC), with the effects of these indicators captured by estimating the p ( and p, parameters. 
The remaining variables are defined as those contained in equation 1. 

While the main effects analysis showed similar performance of students in charter and MPS schools, results 
presented here by charter school type reveal a more nuanced picture. These results are depicted in Table 6. 
Conversion charter school students make significantly greater four-year achievement gains than MPS students 
in reading. Controlling for baseline test scores and student characteristics (Table 6, Model 3), conversion charter 
attendance is estimated to increase four-year reading achievement gains by .193 standard deviations. This 
effect is highly statistically significant at the 99 percent level of confidence. However, in math, the conversion 
charter advantage in Model 2 becomes non-significant in Model 3 once accounting for student background 
characteristics. Students in non-conversion charter schools, in contrast to those in conversion charter schools, 
perform no differently than MPS students across all models for both subjects. 


12 In most states, schools are called "conversion charters" if they are former traditional public schools that converted to charter 
school status. We use the term here to refer to former private schools because a substantial number of private schools (4) have 
converted to public charter status in Milwaukee. Most of the conversion charters in this study were formerly in the Milwaukee 
Parental Choice (voucher) Program. Schools may have switched status because they received greater compensation 
per student as charter schools, and, unlike in the voucher program, student eligibility in charters is not limited to low- 
income families. 
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Table 6. Four- Year Growth Models of Math and Reading Achievement for Conversion 
and Non-Conversion Charter Impacts, 2006-2010 



Model 1- Conversion and 
Non-Conversion Charter 
Status 

Model 2- Controlling for 
Prior Test Scores Only 

Model 3- Student 
Characteristics Included 

Math 

2010 

Reading 

2010 

Math 

2010 

Reading 

2010 

Math 

2010 

Reading 

2010 

Conversion Charter 2006 

. 468 “* 

(. 161 ) 

. 496 “* 

(. 164 ) 

. 213 “ 

(. 086 ) 

. 253 *** 

(. 096 ) 

.113 

(. 070 ) 

. 193 “* 

(. 069 ) 

Non-Conversion Charter 2006 

-.045 

(. 152 ) 

-.143 

(. 153 ) 

.068 

(. 083 ) 

-.010 

(. 104 ) 

.033 

(. 068 ) 

-.011 

(. 087 ) 

2006 Score - Reading 



. 218 *“ 

(. 028 ) 

. 456 "* 

(. 031 ) 

. 191 *" 

(. 027 ) 

. 407 *** 

(. 026 ) 

2006 Score - Math 



. 428 “* 

(. 029 ) 

. 205 "* 

(. 027 ) 

. 443 "* 

(. 031 ) 

^ -| y*** 

(. 026 ) 

Grade 7 





-. 241 " 

(. 109 ) 

-. 283 *“ 

(. 089 ) 

Grade 8 





-.213 

(. 135 ) 

-. 229 “ 

(. 090 ) 

Grade 10 





-. 478 *" 

(. 137 ) 

-. 454 *“ 

(. 106 ) 

Black 





-. 265 *" 

(. 076 ) 

-. 343 *“ 

(. 072 ) 

Female 





-.070 

(. 068 ) 

.073 

(. 055 ) 

ExEd 





-. 224 " 

(. 089 ) 

-. 387 “* 

(. 070 ) 

Free Lunch 





-.062 

(. 055 ) 

-.062 

(. 055 ) 

Constant 

1 3 1 *** 
(. 045 ) 

-.049 
( . 048 ) 

.015 

(. 030 ) 

. 067 " 

(. 028 ) 

. 718 "* 

(. 133 ) 

. 758 *“ 

(. 097 ) 

N 

1556 

1559 

1556 

1559 

1556 

1559 

R 2 

.033 

.046 

.429 

.456 

.463 

.491 

F 

4.37 

5.32 

269.03 

229.50 




***p<0.01, **p<0.05, *p<0.10 


Tabled results control for race, gender, grade, free lunch, exceptional education in Model 3. Reference categories for these variables are 
non-black, male, grade 6 in 2010, no free lunch, and no exceptional education. Two prior test scores are also controlled for in Models 
2 and 3. Robust standard errors are estimated to account for the clustering of students within particular schools. F-tests cannot be 
computed using robust standard errors when a cell in the regression matrix is only a single student. 
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The historical plots of results for conversion and non-conversion charter schools for each year of the 
evaluation are displayed in Figures 4 and 5. Conversion charters show a consistent, upward trend in 
reading achievement gains over time. By contrast, trends over time for conversion charter schools in 
math and non-conversion charter schools in both subjects are more mixed, showing fluctuations in 
achievement gains. 


Figure 4. Effect of Conversion Charter Attendance on Student Achievement, by Year 

Point Estimate and 90% Confidence Interval 
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Figure 5. Effect of Non-Conversion Charter Attendance on Student Achievement, by Year 

Point Estimate and 90% Confidence Interval 
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Variation in Main Effects by Student Achievement Levels 

A second supplemental analysis examines potential variation in charter school impacts based on the level of 
achievement of students in 2010. We do this using quantile regressions, which estimate the effects of being in 
charter schools for students at different levels of achievement. We examine charter impacts at the 10 th , 25 th , 50 th , 
75 th and 90 th percentiles of the 2010 outcome achievement distributions. All of the controls included in Model 
3 in Tables 5 and 6 are included in the quantile regressions. The results for quantile regressions in reading and 
math are presented in Table 7. 


Table 7. Results of Quantile Regressions Estimating Charter Impacts on Four-Year Achievement Gains 



.10 

.25 

Median 

.75 

.90 

Math (N=1 556) 

Charter main effect 

.131 (.089) 

,063(.053) 

.01 8(.045) 

,039(.041) 

,048(.047) 

Conversion charter 

.171 (.166) 

.071 (.079) 

414(.053)** 

,044(.068) 

,090(.079) 

Non-conversion 

charter 

.105(407) 

,059(.057) 

,004(.040) 

-,025(.052) 

-,020(.058) 


Reading (N=1559) 

Charter main effect 

-,008(.092) 

.041 (.048) 

,086(.028)*** 

422(.041)‘‘* 

,026(.055) 

Conversion charter 

.229(445) 

.243 (.071)*** 

449(.051)*** 

462(.058)*** 

,083(.076) 

Non-conversion 

charter 

-413(.099) 

-.01 3(.052) 

,060(.039) 

,080(.046)* 

,014(.065) 


***p<0.01, **p<0.05, *p<0.10 

Note: Test scores measured as standardized scores. Tabled results control for race, free lunch status, exceptional education, grade, 
and prior achievement. The point estimates for these controls are not included in the table but are available upon request. 


Quantile results show that there are few significant effects of independent charter schools on average and these 
effects are confined to reading. Independent charters deliver comparatively higher rates of gain for students in 
reading at the median and at the 75 th percentile of that distribution. The absence of significant charter effects 
in math suggests that independent charter schools perform the same as MPS schools at all points of the 
achievement distribution in math. 

When results are split by the type of charter school, conversion charter schools are more effective for students 
at the median in math and at the 25 th , 50 th and 75 th percentiles of the distribution in reading. The conversion 
charter effects in reading are highly statistically significant at the 99 percent confidence level. Conversion 
charter schools appear to do the most for students at the 25 th percentile in reading who make about .24 standard 
deviations more gains than students in MPS. 
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Variation in Main Effects for Students Who Stay in the Same Sector for Five Years 

The main effects for this study used all students in charter schools and an equal number of matched MPS 
students based on their initial sector attendance in 2006. However, students switch schools and sectors, and 
some are lost to the study. These are ongoing problems for longitudinal observation and random assignment 
studies. A key issue is the attribution of treatment effects. For example, if a student is initially in a charter 
school, but then spends the next three years in an MPS school, how do you attribute their learning gains? 

There is no single answer. We could allow the initial condition to determine sector, as we have in the main 
analysis and as is done in randomized field trials, or we could weight the relative exposure to charters or MPS 
(a dosage effect). 

Another method is to estimate the results only for those students who remain in the same sector for the 
duration of the study. Following this approach, we estimated results for a subgroup of “stayers”. Stayers are 
students who remain in the same sector, independent charter or MPS, for the five years of the study. These 
estimates are a sensitivity analysis to control for attrition and those who crossover between sectors. 

The results for the stayers’ analysis are presented in Tables 8 and 9. Amongst stayers, there are large and 
substantial positive effects for students who remain in independent charter schools between 2006-07 and 2010- 
11. Four-year achievement gains of stayers in independent charter schools are on the order of .411 standard 
deviations in math and .335 standard deviations in reading (Table 8). These effects reach the 99 percent level of 
confidence. Considering that the students in charter schools performed similarly to MPS students in the full 
sample, the effects of charter schools on stayers are very large. 
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Table 8. Non-Sector Switching (Stayer) Four-Year Growth Models of Math and Reading Achievement for Independent 

Charters on Average, 2006-2010 



Model 1- Charter Status 

Model 2- Controlling for 
Prior Test Scores Only 

Model 3- Student 
Characteristics Included 

Math 

2010 

Reading 

2010 

Math 

2010 

Reading 

2010 

Math 

2010 

Reading 

2010 

Charter 2006 

. 573 *** 

(. 122 ) 

. 467 *“ 

(. 126 ) 

. 458 *“ 

(. 063 ) 

^ £- 

(. 074 ) 

. 411 *** 

(. 074 ) 

^ ^ ^ $HHF 

(. 071 ) 

2006 Score - Reading 



. 214 *** 

(. 033 ) 

. 494 “* 

(. 038 ) 

. 170 *** 

(. 033 ) 

. 428 *“ 

(. 037 ) 

2006 Score - Math 



. 426 *** 

(. 040 ) 

. 181 “* 

(. 033 ) 

. 428 *** 

(. 041 ) 

. 188 *“ 

(. 033 ) 

Grade 7 





-.162 

(. 101 ) 

-. 228 “ 

(. 087 ) 

Grade 8 





-.102 

(. 117 ) 

-.127 

(. 087 ) 

Grade 10 





-.199 

(. 123 ) 

-. 224 “ 

(. 093 ) 

Black 





-. 171 ” 

(. 072 ) 

-. 309 *** 

(. 069 ) 

Female 





-.021 

(. 050 ) 

. 103 “ 

(. 047 ) 

ExEd 





-. 296 *** 

(. 091 ) 

-. 386 *“ 

(. 092 ) 

Free Lunch 





-.105 

(. 064 ) 

-. 106 * 

(. 061 ) 

Constant 

-. 103 ** 

(. 046 ) 

-.024 

(. 051 ) 

.018 

(. 032 ) 

. 072 “ 

(. 030 ) 

. 466 *** 

(. 115 ) 

. 605 *** 

(. 107 ) 

N 

1068 

1068 

1068 

1068 

1068 

1068 

R 2 

.082 

.052 

.487 

.505 

.509 

.534 

F 

22.08 

13.85 

157.55 

175.93 




***p<0.01, **p<0.05, *p<0.10 


Tabled results control for race, gender, grade, free lunch, exceptional education in Model 3. Reference categories for these variables are 
non-black, male, grade 6 in 2010, no free lunch, and no exceptional education. Two prior test scores are also controlled for in Models 2 
and 3. Robust standard errors are estimated to account for the clustering of students within particular schools. F statistics could not be 
computed for Model 3 because a regressor in one of the clusters contained only one nonzero value. F-tests cannot be computed using 
robust standard errors when a cell in the regression matrix is only a single student. 
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Table 9. Non-Sector Switching (Stayer) Four-Year Growth Models of Math and Reading Achievement for Conversion and 

Non-Conversion Charter Impacts, 2006-2010 



Model 1- Charter Status 

Model 2- Controlling for 
Prior Test Scores Only 

Model 3- Student 
Characteristics Included 

Math 

2010 

Reading 

2010 

Math 

2010 

Reading 

2010 

Math 

2010 

Reading 

2010 

Conversion Charter 2006 

. 692 *** 

(. 163 ) 

. 679 *** 

(. 131 ) 

. 440 *** 

(. 067 ) 

. 422 *** 

(. 066 ) 

. 362 *** 

(. 111 ) 

. 368 *" 

(. 050 ) 

Non- Conversion 
Charter 2006 

. 455 *** 

(. 140 ) 

.258 

(. 155 ) 

. 475 *** 

(. 095 ) 

. 295 ** 

(. 129 ) 

. 458 *** 

(. 081 ) 

. 302 " 

(. 125 ) 

2006 Score - Reading 



. 214 *** 

(. 033 ) 

. 491 *** 

(. 037 ) 

. 171 *** 

(. 033 ) 

. 427 *" 

(. 037 ) 

2006 Score - Math 



. 426 *** 

(. 040 ) 

. 179 “* 

(. 033 ) 

. 429 *** 

(. 041 ) 

. 187 *** 

(. 033 ) 

Grade 7 





-.158 

(. 098 ) 

-. 231 " 

(. 088 ) 

Grade 8 





-.101 

(. 114 ) 

-.128 

(. 087 ) 

Grade 10 





-.194 

(. 120 ) 

-. 228 “ 

(. 094 ) 

Black 





-. 197 " 

(. 085 ) 

-. 291 *" 

(. 061 ) 

Female 





-.016 

(. 048 ) 

. 100 ” 

(. 047 ) 

ExEd 





-. 292 *** 

(. 092 ) 

-. 389 *** 

(. 092 ) 

Free Lunch 





-. 117 " 

(. 059 ) 

-.097 

(. 060 ) 

Constant 

-. 103 ** 

(. 047 ) 

-.024 

(. 051 ) 

.019 

(. 032 ) 

. 071 “ 

(. 030 ) 

. 494 *** 

(. 124 ) 

. 586 *" 

(. 096 ) 

N 

1068 

1068 

1068 

1068 

1068 

1068 

R 2 

.087 

.069 

.487 

.506 

.509 

.534 

F 

13.11 

14.02 

134.85 

123.86 




***p<0.01, **p<0.05, *p<0.10 


Tabled results control for race, gender, grade, free lunch, exceptional education in Model 3. Reference categories for these variables are 
non-black, male, grade 6 in 2010, no free lunch, and no exceptional education. Two prior test scores are also controlled for in Models 
2 and 3. Robust standard errors are estimated to account for the clustering of students within particular schools. F-tests cannot be 
computed using robust standard errors when a cell in the regression matrix is only a single student. 
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When the effects for stayers are estimated by charter school type, positive results are found for students in 
both conversion and non-conversion charter schools. The effects for students in conversion charter schools are 
considerably larger than for non-conversion charter students in the full sample in both math and reading. In 
math for the full sample, the charter advantage was .11 and not statistically significant (Table 6, Model 3). The 
charter advantage increases to .362 and is significant at the .01 level when we estimate the effect of conversion 
charters on stayers. Reading results are almost double those in the full sample with the point estimate .368 
standard deviations greater than MPS student stayers. A similar, but even more pronounced trend occurs for 
students in non-conversion charter schools if they remain in their sector for five years. Results for these students 
in the full sample (Table 6, Model 3) show no differences between students in non-conversion charters and the 
MPS matched sample. However, for non-conversion stayer students the four-year math achievement gains 
are on the order of .458 standard deviations greater than for stayer students in MPS (Table 9, Model 3). The 
differences in reading go from -.01 (Table 6) to +.302 (Table 9), with the former being essentially zero and the 
latter significant with 99 percent confidence. 


CAVEATS 


In most studies of schools located in poor urban contexts, typically there are missing data due to sample 
attrition. Issues of sample attrition are explored in substantial detail in Appendix A, but we note here that 
about 34 percent of panelists are missing in 2010, with 11.3 percent of MPS sample members missing and 
22.7 percent of independent charter sample members missing. 13 Although these numbers are much lower than 
expected, and lower than in a number of other studies, they could raise concerns that the attrition was non- 
random. In theory, this could affect accurate overall population estimates of gains, but because there were few 
baseline test differences between missing students from either sample, we believe our sample comparisons are 
accurate (See Appendix A). Nevertheless, we weighted the analytic sample to re-balance the sample to reflect 
the equivalence of our matched samples at baseline. Thus we feel confident that attrition has been adequately 
addressed. 

The strong results favoring conversion charter schools and students who remain in the same sector for five years 
raise selection issues. If conversion charter schools entered the study with students more readily able to gain in 
achievement, they would have an advantage over the other charter schools or the MPS sample. We must assume 
that many of the students in 2006 were in these schools before they converted to charter schools, and private 
schools often can more easily select out students. However, an equally plausible alternative is that these may be 
more experienced and effective schools. After all, the conversion charter schools in our sample had operated 
as private schools in the past, meaning they did not have to face the initial learning curve that startup charters 


13 Missing means that we cannot locate a student in any database we use (enrollment, testing, etc.). Those students who "age 
out" of testing, but can still be accounted for, are not counted as attriting from the study. 
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need to negotiate. Similarly, if charter school stayers are differentially selected — either by school or by family 
choices — than their counterpart stayers in MPS, the results would also be biased in favor of charters through 
that selection. However, the alternative is that charter schools may simply have an increasingly more positive 
effect on students the longer they stay in the charter school. 


SUMMARY AND CONCLUSIONS 

This is the final report about the performance of independent charter students in Milwaukee compared to 
a matched sample of students in the Milwaukee Public Schools. The results of this and previous analyses of 
achievement gains generally support the existence of at least some positive independent charter school effects 
over time in Milwaukee. 

The results of the comparisons on math and reading scores differ by years and by the statistics employed. We 
reported last year that there was 

...a clear pattern of positive charter school effects growing over time. There was little consistent 
evidence of differences in achievement gains between charter and MPS students after one year. The 
second year growth was better for charters in some models and for some tests, but not for others. In 
the third year of growth, a sizable independent charter school advantage was apparent in all of our 
analyses (Witte et al.,2§\\). 

That trend was not continued in the fifth year where estimates of four-year achievement growth are positive 
for charter schools but the basic models do not produce statistically significant differences between students 
attending all independent charters and the MPS sample of students. On the other hand, students in conversion 
charter schools, which were once private schools, consistently outperformed similar MPS students in our 
matched sample in every year. In one model, the larger growth in math after five years is positive but not quite 
statistically significant. 

One of our supplementary analyses determined that students who remained in charter schools over five years 
(e.g. “stayers”) made significant achievement gains in both reading and math compared to their counterpart 
stayers in MPS. This finding holds for charter schools on average, as well as conversion and non-conversion 
charter schools. These results are between three- and four- tenths of a standard deviations and significant at 
the 99 percent confidence level. In sum, our study results suggest that a given Milwaukee student stands a 
reasonable chance of achieving higher test scores if the student enrolls in an independent public charter school 
as opposed to an MPS school, especially if the test is in reading, the charter used to be a private school, and the 
student remains at that school for many years. Finally, the fact that non-conversion charters performed about 
as well as conversion charters with stayer students, but performed less well than conversion charters overall, 
suggests that conversion charters outperformed non-conversion charters in our study specifically because they 
kept a higher proportion of their students in their school for the duration. 
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Appendix A- Study Attrition 


Missing and Non-Missing Students 

About 34 percent of students in our sample were 
unable to be located in 2010-11. Missing students 
in the two samples are not the same as non-missing 
students. The relevant data are portrayed in Table 
A-1 . Generally, missing students in 2010-1 1 exhibit 
baseline achievement which is higher in math than 
non-missing students. There are no differences, 
however, in baseline achievement in reading 
between the two samples. Students who are 
missing are more likely than non-missing students 
to be White, Hispanic, in grade 5, or in grade 8 at 
baseline. These students are also less likely to be 
Black, on free lunch, have a disability and be in 
grades 3 or 4 at baseline. It is important to note that 
these differences are not large and are controlled 
for in our statistical models presented in the text of 
this report. 


Table A-1 . Baseline Student Characteristics for 
Non-Missing and Missing Students in 2010-11 



Non-Missing 

Missing 

Students 

Average Mean Baseline Math 

-.152** 

-.083 

Average Mean Baseline Reading 

-.101 

-.050 

Female 

1,465 

779 

(%) 

(49.19) 

(49.94) 

Black 

2,880“ 

1,479 

(%) 

(96.71) 

(94.81) 

White 

67“ 

54 

(%) 

(2.25) 

(3.46) 

Hispanic 

25* 

22 

(%) 

(0.84) 

(1.41) 

Native American 

3 

4 

(%) 

( 0 . 10 ) 

(0.26) 

Asian 

3 

1 

(%) 

( 0 . 10 ) 

(0.06) 

Free Lunch 

2,334“ 

1,116 

(%) 

(78.71) 

(71.54) 

ExEd 

397 *“ 

159 

(%) 

(13.33) 

(10.19) 

Baseline Grade 3 

600"* 

255 

(%) 

(20.15) 

(16.35) 

Baseline Grade 4 

606“ 

224 

(%) 

(20.35) 

(14.36) 

Baseline Grade 5 

522“ 

332 

(%) 

(17.53) 

(21.28) 

Baseline Grade 6 

455 

259 

(%) 

(15.28) 

(16.60) 

Baseline Grade 7 

439 

248 

(%) 

(14.74) 

(15.90) 

Baseline Grade 8 

356“ 

242 

(%) 

(11.95) 

(15.51) 

TOTAL (N) 

2,978 

1,560 


***p<0.01, **p<0.05, *p<0.10 

Note: The average mean baseline math and reading scores are normalized z 
scores. Stars indicate that non missing students are different from missing 
students based on a two-sample difference in proportions test. Percentages 
are rounded to the nearest hundredth. 
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Missing Students Between Sectors Table A-2. MPS vs. Independent Charter Attrition Statistics for 2010-11 


As indicated in Table A-2, there is considerable difference 
in the number of students missing in the two sectors, with 


MPS Matched 

Independent 

Charter 

1 1.2 percent missing in MPS and 22.7 percent missing from 

Average Mean 

-.151* 

-.050 

the independent charters. Students in both independent 

Baseline Math 



charters and MPS may have left for private schools in 

Average Mean 

-.066 

-.043 

the MPCP program or may have moved out of the city of 

Baseline Reading 



Milwaukee. We have tracked students between sectors (i.e. 

Female 

244 

535 

"crossovers") using test score and enrollment data. We do 

(%) 

(47.20) 

(51.29) 

not have the data to track students into private schools, 

Black 

502*** 

977 

which are likely to account for more missing students 

(%) 

(97.10) 

(93.67) 

in the charter schools. Based on the results of previous 

White 

14 

40 

longitudinal studies conducted in urban contexts, we 

(%) 

(2.71) 

(3.84) 

anticipated sample attrition at 20 percent annually. As 

Hispanic 

j *** 

21 

a result, we are pleased with the relatively low level of 

(%) 

(0.19) 

(2.01) 

sample attrition. 

Native American 

0 

4 

Students who are missing in the two sectors for 2010-1 1 

(%) 

(0.00) 

(0.38) 

differ on some student characteristics. Compared to MPS 

Asian 

1 

1 

students, independent charter students are less likely to 

(%) 

(0.10) 

(0.06) 

be Black and in grade 3 at baseline. However, recall that 

Free Lunch 

378 

738 

the charter sample is overwhelmingly black to begin 

(%) 

(73.11) 

(70.76) 

with (Table C2) so few students can comprise the racial 

ExEd 

74*** 

85 

difference. These same students are more likely to be 

(%) 

(14.31) 

(8.15) 

Hispanic and in grade 6 at baseline (Table A-2). In addition, 

Baseline Grade 3 

101** 

154 

these students are more likely to score higher in math on 

(%) 

(19.54) 

(14.77) 

average than their MPS counterparts, but there are no 

Baseline Grade 4 

76 

148 

differences in reading. 

(%) 

(14.70) 

(14.19) 


Baseline Grade 5 

102 

230 


(%) 

(19.73) 

(22.05) 


Baseline Grade 6 

73* 

186 


(%) 

(14.12) 

(17.83) 


Baseline Grade 7 

87 

161 


(%) 

(16.83) 

(15.44) 


Baseline Grade 8 

78 

164 


(%) 

(15.09) 

(15.72) 


TOTAL (N) 




(%) 

517 

1,043 


***p<0.01, **p<0.05, *p<0.10 

Note: The average mean baseline math and reading scores are 
normalized z scores. Stars indicate MPS Matched different from 
Independent Charter statistics based on a two-sample difference in 
proportions test. Percentages are rounded to the nearest hundredth. 
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Appendix B- School Switching 

The explanatory results for models including school switching are described below. We view school switching as a downstream mediator meaning 
that switching can be considered as a part of the treatment (independent charter school attendance). When we include school switching the 
charter main effects in math and reading continue to be non-significant, providing evidence that these students perform similarly to MPS students 
in this model (Table B-1, full sample). Estimating the models by charter school type, we find that the conversion charter effect becomes non- 
significant in reading after including school switching (Table B-2, full sample). Four-year math achievement gains for conversion charters and four- 
year achievement gains in both subjects for non-conversion charter schools remain non-significant, showing these students perform no differently 
from MPS students (Table B-2, full sample). 

Results from regressions of stayers, students who remain in the same sector over the five years of the study, show some significant advantages for 
charter students after controlling for school switching. Specifically, charter students on average as well as conversion and non-conversion charter 
students separately, make greater four-year gains than MPS students in math (See Tables B-1 and B-2, stayers sample). Similarly, students in 
charter schools on average and conversion charters make higher four-year reading achievement gains. 

However, switching schools among charter stayers is limited to 20 students, with only 2 students switching from conversion charter schools. What 
this may mean is that where there are positive effects of independent charter schools on average and by type, they may be partially explained 
by the greater stability of charter attendance after four years. Specifically, there are many fewer charter school students switching schools after 
four years. This could be an effect of family desire to remain in the specific charter school their child attends, a preference for independent charter 
schools in general, or that there are fewer switching choices available to them than to parents in MPS schools. 
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Model 4- Switching Estimates for 
Full Sample 

Model 4- Non-Sector Switching 
Estimates for Stayers Sample 


Math 

2010 

Reading 

2010 

Math 

2010 

Reading 

2010 

Charter 2006 

-.022 

-.020 

. 303 *** 

. 193 ** 

(. 042 ) 

(. 055 ) 

(. 083 ) 

(. 074 ) 

2006 Score - Reading 

. 183 *** 

. 402 *** 

. 170 *** 

. 428 *** 

(. 026 ) 

(. 026 ) 

(. 033 ) 

(. 036 ) 

2006 Score - Math 

. 424 *** 

. 204 *** 

. 422 *** 

. 181 *** 

(. 030 ) 

(. 023 ) 

(. 042 ) 

(. 032 ) 

Grade 7 

-. 293 *** 

-. 324 *** 

-. 185 * 

-. 260 *** 

(. 101 ) 

(. 100 ) 

(. 102 ) 

(. 088 ) 

Grade 8 

-. 279 ** 

-. 291 *** 

-.126 

-. 159 * 

(. 123 ) 

(. 093 ) 

(. 119 ) 

(. 088 ) 

Grade 10 

-. 399 *** 

-. 381 *** 

-.193 

-. 217 ** 

(. 119 ) 

(. 103 ) 

(. 125 ) 

(. 093 ) 

Black 

-. 207 *** 

-. 327 *** 

-. 165 ** 

-. 301 *** 

(. 069 ) 

(. 075 ) 

(. 071 ) 

(. 070 ) 

Female 

-.076 

.075 

-.022 

. 101 ** 

(. 068 ) 

(. 058 ) 

(. 050 ) 

(. 047 ) 

ExEd 

-. 230 *** 

-. 383 *** 

-. 295 *** 

-. 384 *** 

(. 084 ) 

(. 072 ) 

(. 091 ) 

(. 092 ) 

Free Lunch 

-. 085 * 

-. 105 ** 

-.102 

-. 103 * 

(. 050 ) 

(. 051 ) 

(. 064 ) 

(. 060 ) 

Switch School 

-. 383 *** 

-. 348 *** 

-. 159 ** 

-. 208 *** 

(. 075 ) 

(. 047 ) 

(. 079 ) 

(. 067 ) 

Constant 

1 . 01 *** 

1 . 07 *** 

. 602 *** 

. 785 *** 

(. 133 ) 

(. 115 ) 

(. 131 ) 

(. 121 ) 

N 

1556 

1559 

1068 

1068 

R 2 

.486 

.506 

.512 

.539 

F 






***p<0,01, **p<0.05, *p<0.10 


Table results control for race, gender, grade, free lunch, exceptional education and switching schools. Reference 
categories for these variables are non-black, male, grade is 6 in 2010 for four-year gains model, no free lunch, 
no exceptional education and not switching schools, respectively. Two prior test scores are also controlled for in 
these models. Robust standard errors are estimated to account for the clustering of students within particular 
schools. Estimates for student characteristics, prior test scores and grade dummies are available upon request. 
F-tests cannot be computed using robust standard errors when a cell in the regression matrix is only a single 
student. 
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B-2. Estimates of Four- Year Gains for Conversion and Non-Conversion Charter Students in the 
Full Sample and the Stayers Sample Controlling for Switching from Initial School 



Model 4- Switching 
Estimates for Full Sample 

Model 4- Non-Sector 
Switching Estimates for 
Stayers Sample 

Math 

2010 

Reading 

2010 

Math 

2010 

Reading 

2010 

Conversion Charter 

-.057 

.051 

.243** 

219." 

2006 

(.059) 

(.064) 

(.115) 

(.058) 

Non- Conversion 

-.009 

-.046 

.353“* 

.170 

Charter 2006 

(.048) 

(.076) 

(.091) 

(.117) 

2006 Score -Reading 

.184“* 

.400*" 

.171*** 

428". 


(.026) 

(.025) 

(.033) 

(.036) 

2006 Score- 

425*** 

203*** 

423*** 

.180*" 

Math 

(.031) 

(.024) 

(.042) 

(.032) 

Grade 7 

-.293*“ 

-.325"* 

-.181* 

-.262*" 


(.100) 

(.099) 

(.099) 

(.089) 

Grade 8 

-.282“ 

285". 

-.126 

-.159* 


(.123) 

(.092) 

(.116) 

(.088) 

Grade 10 

-.398*“ 

385". 

-.186 

-.220“ 


(.118) 

(.103) 

(.121) 

(.094) 

Black 

_ 219**. 

-.304"* 

-.194“ 

-.288"* 


(.065) 

(.072) 

(.084) 

(.062) 

Female 

-.074 

.071 

-.017 

.098“ 


(.066) 

(.055) 

(.048) 

(.046) 

ExEd 

-.226“ 

-390". 

-.290*** 

387". 


(.086) 

(.070) 

(.092) 

(.092) 

Free Lunch 

-.093“ 

-.087* 

-.116“ 

-.096 


(.048) 

(.050) 

(.059) 

(.060) 

Switch School 

-.392*“ 

329". 

-.164“ 

-205". 


(.079) 

(.054) 

(.080) 

(.066) 

Constant 

1 .03“* 

1.02*“ 

.639“* 

769... 


(.140) 

(.122) 

(.139) 

(.114) 

N 

1556 

1559 

1068 

1068 

R 2 

.486 

.506 

.513 

.539 

F 






***p<0.01, **p<0.05, *p<0.10 


Tabled results control for race, gender, grade, free lunch, exceptional education and switching schools. 
Reference categories for these variables are non-black, male, grade is 6 in 2010 for four-year gains 
model, no free lunch, no exceptional education and not switching schools, respectively. Two prior test 
scores are also controlled for in these models. Robust standard errors are estimated to account for the 
clustering of students within particular schools. Estimates for student characteristics, prior test scores 
and grade dummies are available upon request. F-tests cannot be computed using robust standard 
errors when a cell in the regression matrix is only a single student. 
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Appendix C- Matching Algorithm 

The first step in our analysis was to determine the comparative samples of students. Because the total number of students in independent charters for 
which test scores were available in 2006-07 was 2,295, we decided to include all of those students in the charter school sample. The issue was then 
how to create a relevant matched sample that would 

be similar on important observed characteristics at Figure Cl . Comparison of Baseline Reading Scale Score 

baseline. To do that we first selected a random sample 
of MPS students matched by grade. In doing so we 
discovered that the baseline test scores (November 
2006) for that group differed from those in the 
independent charter schools in a number of grades. As 
depicted in Figures Cl and C2, the random MPS sample 
of students usually scored higher in 2006 than the 
independent charter students. Thus the random sample 
would have started out students at different levels of 
prior achievement. 

To adjust for this problem we undertook a two-step 
procedure. First, each student in the charter panel was 
matched with the set of MPS students in their grade 
with baseline WKCE test scores within five percent of 
their score. There were 20 such bands with the lowest 
being scores from the first to fifth percentile, and the 
highest for those scoring from 95% to 100%. Second, 
the charter panelist and each MPS student within that 
five percent grade band were assigned a propensity 
score that predicted their likelihood of being in a 
charter school based on race, gender, English Language 
Learner (ELL) status, and participation in the federal 
Free/Reduced Price Lunch (FRL) program. The MPS 
student within the grade band with the charter school 
propensity score closest to the propensity score of a 
given student in the charter panel was drawn out of 
the panel (without replacement) to serve on the MPS 
comparison panel. The result of all these matches was a 
panel of 2,295 independent charter school students and 
2,295 MPS comparison students that closely resemble 
the charter school students on baseline test scores and 
other factors that predict charter school enrollment. 

The purpose of this procedure was to reduce the 
differences in observed characteristics between the 

independent charter students and a random sample of MPS students. Figures Cl and C2 and Table Cl depict our success in improving on the extent to 
which our charter and comparison groups are carefully matched. These figures and table display mean comparisons by grade level for the three possible 
comparison samples. 


Figure C2. Comparison of Baseline Math Scale Score 
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In Figure Cl, for reading, the Independent Charter Sample and MPS Matched Sample begin at similar points and converge in later grades. While 
there appears to be a gap over grades 5 through 7, these mean differences between the two sectors are not statistically significant. For math, in 
Figure C2, the charter and MPS matched samples are almost indistinguishable at all grades. 

In Table Cl, we see few statistically significant differences between means in reading and math scale scores when comparing the Independent 
Charter Sample and the MPS Matched Sample. With the exceptions of 4 th and 6 th grade math, this holds for every grade in both subjects. Both of 
those lingering differences between the charter and matched samples are statistically significant only at the 90 percent confidence level, the lowest 
confidence level that we use in this evaluation. This suggests the matching was successful. Thus, in terms of prior achievement we have created 
samples that are essentially equal at baseline. 


Table Cl . Baseline (2006-07) Scale Scores of Independent Charter, MPS Matched, and MPS Random Samples 


Grade 

Sample 

N 

Mean 
Reading 
Scale Score 

Reading 
SD - 

Mean Math 
Scale Score 

Math SD 

Comb Mean 
Scale Score 

Combined 

SD 

3 

Independent 

430 

429 

47 

385 

49 

407 

44 

3 

MPS Matched 

431 

427 

47 

383 

47 

405 

42 

3 

MPS Random 

341 

440*** 

38 

402*** 

50 

422*** 

40 

4 

Independent 

417 

441 

49 

421 

48 

431 

45 

4 

MPS Matched 

420 

440 

53 

415* 

54 

428 

50 

4 

MPS Random 

324 

447 

53 

434 *** 

47 

441*** 

47 

5 

Independent 

427 

447 

51 

452 

48 

450 

46 

5 

MPS Matched 

430 

451 

49 

454 

41 

453 

40 

5 

MPS Random 

338 

448 

56 

452 

44 

451 

47 

6 

Independent 

358 

460 

51 

473 

48 

466 

46 

6 

MPS Matched 

356 

464 

50 

467* 

40 

466 

41 

6 

MPS Random 

330 

464 

54 

469 

44 

468 

44 

7 

Independent 

346 

470 

51 

493 

43 

482 

44 

7 

MPS Matched 

341 

470 

45 

496 

38 

483 

38 

7 

MPS Random 

303 

468 

49 

499 

47 

488* 

44 

8 

Independent 

298 

499 

47 

507 

48 

503 

43 

8 

MPS Matched 

299 

499 

50 

506 

45 

503 

43 

8 

MPS Random 

290 

488** 

55 

497** 

50 

493** 

53 


***Different from Independent Charter at p<0.01, **Different from Independent Charter at p<0.05, 
* Different from I ndependent Charter at p<0. 10 
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Table C2 indicates that our matching algorithm was successful in producing a matched sample with important measurable student characteristics 
that, though imperfect, are closer than in the random sample. The matched and charter samples are very close on race and gender. Although 
the matched MPS sample contains populations of exceptional education and free lunch students that are closer to the charter sample than the 
random sample provided, the MPS Matched sample does differ significantly from the Independent Charter Sample regarding these two student 
characteristics. 14 


Table C2. Comparison of Student Characteristics in Three Possible Study Samples 


Sample 

Black 

(%) 

White 

(%) 

Hispanic 

(%) 

Female 

(%) 

ExEd 

(%) 

Federal 

Lunch 

(%) 

ELL 

(%) 

Independent 

Charter 

93.93 

3.86 

1.80 

49.67 

8.38 

51.64 

0.04 

MPS Matched 

98.19*** 

1.44*** 

0.26*** 

49.45 

16.07*** 

73.82*** 

0.04 

MPS Random 

59.91*** 

14.52*** 

19.76*** 

48.29 

19.47*** 

83.68*** 

13.27“* 


Stars indicate MPS Matched and MPS Random different from Independent Charter at***p<0.01, **p<0.05, *p<0.10, based on a 
two-tailed t-test. 


14 The initial difference between the charter and matched sample on free lunch status is due to incomplete free lunch data counts 
in a few schools. We correct for this in our models in two ways. If a student had a free lunch observation in 2007-08, 2008-09 
or 2009-1 0, we back-filled the data. In addition, for students with missing data on free-lunch or any other control variable, we 
include an indicator in our models controlling for this missing data. Doing so allows our regression models to draw upon the 
actual data in each student observation, and only that actual data, to inform the coefficient estimates of the model (Cohen and 
Cohen 1983). 
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